Of Short Lists, Pandas and Gibbons

• From human to data-driven decision-making:
  • Black-box models like deep neural networks are being deployed virtually everywhere.
  • More likely than not that your loan or employment application is handled by an algorithm.
  • Includes critical domains: health care, autonomous driving, finance, …

“Weapons of Math Destruction”

• If left unchallenged, these properties of black-box models can create undesirable dynamics:
  • Human operators in charge of the system have to rely on it blindly.
  • Those individuals subject to it generally have no way to challenge an outcome.

“You cannot appeal to (algorithms). They do not listen. Nor do they bend.”

— Cathy O’Neil in Weapons of Math Destruction, 2016

Figure 2: Cathy O’Neil. Source: Cathy O’Neil

Towards Trustworthy AI

Towards Trustworthy AI

Model objective: maximize \(p(\mathcal{D}|\theta)\) where \(\mathcal{D}=\{(x,y)\}_{i=1}^n\) (supervised)

Towards Trustworthy AI

A Framework for Counterfactual Explanations

• Objective originally proposed by Wachter, Mittelstadt, and Russell (2017) is as follows where \(h\) relates to the complexity of the counterfactual and \(M\) denotes the classifier:

\[ \min_{x\prime \in \mathcal{X}} h(x\prime) \ \ \ \mbox{s. t.} \ \ \ M(x\prime) = t \qquad(1)\]

• Typically approximated through regularization:

\[ x\prime = \arg \min_{x\prime} \ell(M(x\prime),t) + \lambda h(x\prime) \qquad(2)\]

Toy example

A simple counterfactual path from 🐱 to 🐶

We have fitted some black-box classifier to divide cats and dogs. One 🐱 is friends with a lot of cool 🐶 and wants to remain part of that group. The counterfactual path below shows her how to fool the classifier:

Counterfactuals … as in Adversarial Examples?

Yes and no!

While both are methodologically very similar, adversarial examples are meant to go undetected while CEs ought to be meaningful.

• Effective counterfactuals should meet certain criteria ✅

• closeness: the average distance between factual and counterfactual features should be small (Wachter, Mittelstadt, and Russell (2017))

• actionability: the proposed feature perturbation should actually be actionable (Ustun, Spangher, and Liu (2019), Poyiadzi et al. (2020))

• plausibility: the counterfactual explanation should be plausible to a human (Joshi et al. (2019))

• unambiguity: a human should have no trouble assigning a label to the counterfactual (Schut et al. (2021))

• sparsity: the counterfactual explanation should involve as few individual feature changes as possible (Schut et al. (2021))

• robustness: the counterfactual explanation should be robust to domain and model shifts (Upadhyay, Joshi, and Lakkaraju (2021))

• diversity: ideally multiple diverse counterfactual explanations should be provided (Mothilal, Sharma, and Tan (2020))

• causality: counterfactual explanations reflect the structural causal model underlying the data generating process (Karimi et al. (2020), Karimi, Schölkopf, and Valera (2021))

Counterfactuals … as in Causal Inference?

NO!

Probabilistic Methods for Counterfactual Explanations

When people say that counterfactuals should look realistic or plausible, they really mean that counterfactuals should be generated by the same Data Generating Process (DGP) as the factuals:

\[ x\prime \sim p(x) \]

• How do we estimate \(p(x)\)? Two probabilistic approaches …

Limited Software Availability

• Some of the existing approaches scattered across different GitHub repositories (🐍).
• Only one unifying Python 🐍 library: CARLA (Pawelczyk et al. 2021).
  • Comprehensive and (somewhat) extensible …
  • … but not language-agnostic and some desirable functionality not supported.
  • Also not composable: each generator is treated as different class/entity.
• Both R and Julia lacking any kind of implementation. Until now …

Enter: CounterfactualExplanations.jl 📦

Package Architecture

Modular, composable, scalable!

using CounterfactualExplanations, Plots, GraphRecipes
plt = plot(AbstractGenerator, method=:tree, fontsize=10, nodeshape=:rect, size=(1000,700))
savefig(plt, joinpath(www_path,"generators.png"))

plt = plot(AbstractFittedModel, method=:tree, fontsize=10, nodeshape=:rect, size=(1000,700))
savefig(plt, joinpath(www_path,"models.png"))

For more detail, check out the repo directly

Generic Generator

# Data:
using CounterfactualExplanations.Data
Random.seed!(1234)
N = 25
xs, ys = Data.toy_data_linear(N)
X = hcat(xs...)
counterfactual_data = CounterfactualData(X,ys')

# Model
using CounterfactualExplanations.Models: LogisticModel, probs 
# Logit model:
w = [1.0 1.0] # true coefficients
b = 0
M = LogisticModel(w, [b])

# Randomly selected factual:
Random.seed!(123)
x = select_factual(counterfactual_data,rand(1:size(X)[2]))
y = round(probs(M, x)[1])
target = ifelse(y==1.0,0.0,1.0) # opposite label as target

# Counterfactual search:
generator = GenericGenerator()
counterfactual = generate_counterfactual(x, target, counterfactual_data, M, generator)

Greedy Generator

# Model:
using LinearAlgebra
Σ = Symmetric(reshape(randn(9),3,3).*0.01 + UniformScaling(1)) # MAP covariance matrix
μ = hcat(b, w)
M = CounterfactualExplanations.Models.BayesianLogisticModel(μ, Σ)

# Counterfactual search:
generator = GreedyGenerator(;δ=0.1,n=25))
counterfactual = generate_counterfactual(x, target, counterfactual_data, M, generator)

REVISE Generator

# Model:
using LinearAlgebra
Σ = Symmetric(reshape(randn(9),3,3).*0.01 + UniformScaling(1)) # MAP covariance matrix
μ = hcat(b, w)
M = CounterfactualExplanations.Models.BayesianLogisticModel(μ, Σ)

# Counterfactual search:
generator = GreedyGenerator(;δ=0.1,n=25))
counterfactual = generate_counterfactual(x, target, counterfactual_data, M, generator)

A more involved example

But things can go wrong …

Custom Models and Interoperability

using Flux, RCall
using CounterfactualExplanations, CounterfactualExplanations.Models
import CounterfactualExplanations.Models: logits, probs # import functions in order to extend

# Step 1)
struct TorchNetwork <: Models.AbstractFittedModel
    model::Any
end

# Step 2)
function logits(M::TorchNetwork, X::AbstractArray)
  nn = M.model
  ŷ = rcopy(R"as_array($nn(torch_tensor(t($X))))")
  ŷ = isa(ŷ, AbstractArray) ? ŷ : [ŷ]
  return ŷ'
end
probs(M::TorchNetwork, X::AbstractArray)= σ.(logits(M, X))
M = TorchNetwork(R"model")

import CounterfactualExplanations.Generators: ∂ℓ
using LinearAlgebra

# Countefactual loss:
function ∂ℓ(generator::AbstractGradientBasedGenerator, counterfactual_state::CounterfactualState) 
  M = counterfactual_state.M
  nn = M.model
  x′ = counterfactual_state.x′
  t = counterfactual_state.target_encoded
  R"""
  x <- torch_tensor($x′, requires_grad=TRUE)
  output <- $nn(x)
  obj_loss <- nnf_binary_cross_entropy_with_logits(output,$t)
  obj_loss$backward()
  """
  grad = rcopy(R"as_array(x$grad)")
  return grad
end

Custom Generators

# Abstract suptype:
abstract type AbstractDropoutGenerator <: AbstractGradientBasedGenerator end

# Constructor:
struct DropoutGenerator <: AbstractDropoutGenerator
    loss::Symbol # loss function
    complexity::Function # complexity function
    mutability::Union{Nothing,Vector{Symbol}} # mutibility constraints 
    λ::AbstractFloat # strength of penalty
    ϵ::AbstractFloat # step size
    τ::AbstractFloat # tolerance for convergence
    p_dropout::AbstractFloat # dropout rate
end

# Instantiate:
using LinearAlgebra
generator = DropoutGenerator(
    :logitbinarycrossentropy,
    norm,
    nothing,
    0.1,
    0.1,
    1e-5,
    0.5
)

import CounterfactualExplanations.Generators: generate_perturbations, ∇
using StatsBase
function generate_perturbations(generator::AbstractDropoutGenerator, counterfactual_state::CounterfactualState)
    𝐠ₜ = ∇(generator, counterfactual_state) # gradient
    # Dropout:
    set_to_zero = sample(1:length(𝐠ₜ),Int(round(generator.p_dropout*length(𝐠ₜ))),replace=false)
    𝐠ₜ[set_to_zero] .= 0
    Δx′ = - (generator.ϵ .* 𝐠ₜ) # gradient step
    return Δx′
end

JuliaCon 2022 and beyond

• To be submitted to JuliaCon 2022 (today 🫣)
• Through the help of community contribution we hope to add:
  • native support for deep learning models (Flux, torch, tensorflow) and other differentiable models.
  • support for non-differentiable models.
  • more generators (DiCE (Mothilal, Sharma, and Tan 2020), ROAR (Upadhyay, Joshi, and Lakkaraju 2021), MINT (Karimi, Schölkopf, and Valera 2021), CLUE (Antorán et al. 2020))

More Resources

• Introductory [blog post: [TDS, homepage]
• Package docs with lots of examples
• Blog post on effortless Bayesian deep learning [TDS, homepage]
• Get in touch!

Explainable AI (XAI)

• interpretable = inherently interpretable model, no extra tools needed (GLM, decision trees, rules, …) (Rudin 2019)
• explainable = inherently not interpretable model, but explainable through XAI

Post-hoc Explainability:

• Local surrogate explainers like LIME and Shapley: useful and popular, but …
  • … can be easily fooled (Slack et al. 2020)
  • … rely on reasonably interpretable features.
  • … rely on the concept of fidelity.
• Counterfactual explanations explain how inputs into a system need to change for it to produce different decisions.
  • Always full-fidelity, since no proxy involved.
  • Intuitive interpretation and straight-forward implemenation.
  • Works well with Bayesian models. Clear link to Causal Inference.
  • Does not rely on interpretable features.
• Realistic and actionable changes can be used for the purpose of algorithmic recourse.

Feature Constraints

counterfactual_data = CounterfactualData(X,ys';domain=[(-Inf,Inf),(-Inf,-0.5)])

Research Topics (1) - Student Project

What happens once AR has actually been implemented? 👀

Research Topics (2)